Undeprecate `ArrowWriter::into_serialized_writer` and add docs #8621

alamb · 2025-10-15T11:05:04Z

Which issue does this PR close?

We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.

Related to Release arrow-rs / parquet Major version 57.0.0 (October 2025) #7835

Rationale for this change

While testing the arrow 57 upgrade in DataFusion I found a few things that need to be fixed
in parquet-rs.

[WIP] Upgrade to arrow/parquet 57.0.0 datafusion#17888

One was that the method ArrowWriter::into_serialized_writer was deprecated, (which I know I suggested in #8389 🤦 ). However, when testing it turns out that the constructor of SerializedFileWriter does a lot of work (like creating the parquet schema from the arrow schema and messing with metadata)

arrow-rs/parquet/src/arrow/arrow_writer/mod.rs

Lines 230 to 263 in c4f0fc1

    
           let mut props = options.properties; 
        
           let schema = if let Some(parquet_schema) = options.schema_descr { 
        
               parquet_schema.clone() 
        
           } else { 
        
               let mut converter = ArrowSchemaConverter::new().with_coerce_types(props.coerce_types()); 
        
               if let Some(schema_root) = &options.schema_root { 
        
                   converter = converter.schema_root(schema_root); 
        
               } 
        
               converter.convert(&arrow_schema)? 
        
           }; 
        
           if !options.skip_arrow_metadata { 
        
               // add serialized arrow schema 
        
               add_encoded_arrow_schema_to_metadata(&arrow_schema, &mut props); 
        
           } 
        
           let max_row_group_size = props.max_row_group_size(); 
        
           let props_ptr = Arc::new(props); 
        
           let file_writer = 
        
               SerializedFileWriter::new(writer, schema.root_schema_ptr(), Arc::clone(&props_ptr))?; 
        
           let row_group_writer_factory = 
        
               ArrowRowGroupWriterFactory::new(&file_writer, arrow_schema.clone()); 
        
           Ok(Self { 
        
               writer: file_writer, 
        
               in_progress: None, 
        
               arrow_schema, 
        
               row_group_writer_factory, 
        
               max_row_group_size, 
        
           })

Creating a RowGroupWriterFactory directly would involve a bunch of code duplication

What changes are included in this PR?

So let's not deprecate this method for now and instead add some additional docs to guide people to the right lace

Are these changes tested?

I tested manually upstream

Are there any user-facing changes?

If there are user-facing changes then we may require documentation to be updated before approving the PR.

If there are any breaking changes to public APIs, please call them out.

alamb · 2025-10-15T11:08:03Z

parquet/src/arrow/arrow_writer/mod.rs

+    /// Flushes any outstanding data before returning.
+    ///
+    /// This can be useful to provide more control over how files are written, for example
+    /// to write columns in parallel. See the example on [`ArrowColumnWriter`].


I tried to drop a few breadcrumbs to find @adamreeve 's new example in #8582

adamreeve · 2025-10-15T17:15:03Z

parquet/src/arrow/arrow_writer/mod.rs

+///
+/// You can create this structure via an [`ArrowRowGroupWriterFactory`]
+///
+/// See the example on [`ArrowColumnWriter`] for how to encode columns in parallel


ArrowRowGroupWriter is actually not public and not used in the ArrowColumnWriter example, so this last sentence might be a little confusing.

(ArrowRowGroupWriterFactory is internally used to create ArrowRowGroupWriters, but publicly it only exposes the ability to create a Vec<ArrowColumnWriter>. If I was starting this again from scratch I might have named things a little differently...)

Thanks -- I tried to improve the comments in 619210d

…_writer

alamb · 2025-10-16T16:11:53Z

Thank you @adamreeve and @mbrobbel

Undeprecate ArrowWriter::into_serialized_writer and add docs

c4f0fc1

github-actions bot added the parquet Changes to the parquet crate label Oct 15, 2025

alamb mentioned this pull request Oct 15, 2025

Unify API for writing column chunks / row groups in parallel #8582

Merged

alamb commented Oct 15, 2025

View reviewed changes

alamb marked this pull request as ready for review October 15, 2025 13:01

mbrobbel approved these changes Oct 15, 2025

View reviewed changes

mbrobbel added this to the 57.0.0 milestone Oct 15, 2025

adamreeve approved these changes Oct 15, 2025

View reviewed changes

alamb added 2 commits October 16, 2025 11:37

Improve comments

619210d

Merge remote-tracking branch 'apache/main' into alamb/into_serialized…

cdfe382

…_writer

alamb merged commit 5a384f4 into apache:main Oct 16, 2025
16 checks passed

alamb deleted the alamb/into_serialized_writer branch October 16, 2025 16:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Undeprecate `ArrowWriter::into_serialized_writer` and add docs #8621

Undeprecate `ArrowWriter::into_serialized_writer` and add docs #8621

alamb commented Oct 15, 2025 •

edited

Loading

Uh oh!

alamb Oct 15, 2025

Uh oh!

adamreeve Oct 15, 2025

Uh oh!

alamb Oct 16, 2025

Uh oh!

alamb commented Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	let mut props = options.properties;

	let schema = if let Some(parquet_schema) = options.schema_descr {
	parquet_schema.clone()
	} else {
	let mut converter = ArrowSchemaConverter::new().with_coerce_types(props.coerce_types());
	if let Some(schema_root) = &options.schema_root {
	converter = converter.schema_root(schema_root);
	}

	converter.convert(&arrow_schema)?
	};

	if !options.skip_arrow_metadata {
	// add serialized arrow schema
	add_encoded_arrow_schema_to_metadata(&arrow_schema, &mut props);
	}

	let max_row_group_size = props.max_row_group_size();

	let props_ptr = Arc::new(props);
	let file_writer =
	SerializedFileWriter::new(writer, schema.root_schema_ptr(), Arc::clone(&props_ptr))?;

	let row_group_writer_factory =
	ArrowRowGroupWriterFactory::new(&file_writer, arrow_schema.clone());

	Ok(Self {
	writer: file_writer,
	in_progress: None,
	arrow_schema,
	row_group_writer_factory,
	max_row_group_size,
	})

Undeprecate ArrowWriter::into_serialized_writer and add docs #8621

Undeprecate ArrowWriter::into_serialized_writer and add docs #8621

Conversation

alamb commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

adamreeve Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Undeprecate `ArrowWriter::into_serialized_writer` and add docs #8621

Undeprecate `ArrowWriter::into_serialized_writer` and add docs #8621

alamb commented Oct 15, 2025 •

edited

Loading